{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "%%shell\n",
        "# Installs the latest dev build of TVM from PyPI. If you wish to build\n",
        "# from source, see https://tvm.apache.org/docs/install/from_source.html\n",
        "pip install apache-tvm --pre"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "(tutorial-deploy-model-on-rasp)=\n",
        "# Deploy the Pretrained Model on Raspberry Pi\n",
        "**Author**: [Ziheng Jiang](https://ziheng.org/),             [Hiroyuki Makino](https://makihiro.github.io/)\n",
        "\n",
        "This is an example of using Relay to compile a ResNet model and deploy\n",
        "it on Raspberry Pi.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import tvm\n",
        "from tvm import te\n",
        "import tvm.relay as relay\n",
        "from tvm import rpc\n",
        "from tvm.contrib import utils, graph_executor as runtime\n",
        "from tvm.contrib.download import download_testdata"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n",
        "## Build TVM Runtime on Device\n",
        "\n",
        "The first step is to build the TVM runtime on the remote device.\n",
        "\n",
        "<div class=\"alert alert-info\"><h4>Note</h4><p>All instructions in both this section and next section should be\n",
        "  executed on the target device, e.g. Raspberry Pi. And we assume it\n",
        "  has Linux running.</p></div>\n",
        "\n",
        "Since we do compilation on local machine, the remote device is only used\n",
        "for running the generated code. We only need to build tvm runtime on\n",
        "the remote device.\n",
        "\n",
        "```bash\n",
        "git clone --recursive https://github.com/apache/tvm tvm\n",
        "cd tvm\n",
        "mkdir build\n",
        "cp cmake/config.cmake build\n",
        "cd build\n",
        "cmake ..\n",
        "make runtime -j4\n",
        "```\n",
        "After building runtime successfully, we need to set environment varibles\n",
        "in :code:`~/.bashrc` file. We can edit :code:`~/.bashrc`\n",
        "using :code:`vi ~/.bashrc` and add the line below (Assuming your TVM\n",
        "directory is in :code:`~/tvm`):\n",
        "\n",
        "```bash\n",
        "export PYTHONPATH=$PYTHONPATH:~/tvm/python\n",
        "```\n",
        "To update the environment variables, execute :code:`source ~/.bashrc`.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Set Up RPC Server on Device\n",
        "To start an RPC server, run the following command on your remote device\n",
        "(Which is Raspberry Pi in our example).\n",
        "\n",
        "```bash\n",
        "python -m tvm.exec.rpc_server --host 0.0.0.0 --port=9090\n",
        "```\n",
        "If you see the line below, it means the RPC server started\n",
        "successfully on your device.\n",
        "\n",
        "```bash\n",
        "INFO:root:RPCServer: bind to 0.0.0.0:9090\n",
        "```\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Prepare the Pre-trained Model\n",
        "Back to the host machine, which should have a full TVM installed (with LLVM).\n",
        "\n",
        "We will use pre-trained model from\n",
        "[MXNet Gluon model zoo](https://mxnet.apache.org/api/python/gluon/model_zoo.html).\n",
        "You can found more details about this part at tutorial `tutorial-from-mxnet`.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from mxnet.gluon.model_zoo.vision import get_model\n",
        "from PIL import Image\n",
        "import numpy as np\n",
        "\n",
        "# one line to get the model\n",
        "block = get_model(\"resnet18_v1\", pretrained=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "In order to test our model, here we download an image of cat and\n",
        "transform its format.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "img_url = \"https://github.com/dmlc/mxnet.js/blob/main/data/cat.png?raw=true\"\n",
        "img_name = \"cat.png\"\n",
        "img_path = download_testdata(img_url, img_name, module=\"data\")\n",
        "image = Image.open(img_path).resize((224, 224))\n",
        "\n",
        "\n",
        "def transform_image(image):\n",
        "    image = np.array(image) - np.array([123.0, 117.0, 104.0])\n",
        "    image /= np.array([58.395, 57.12, 57.375])\n",
        "    image = image.transpose((2, 0, 1))\n",
        "    image = image[np.newaxis, :]\n",
        "    return image\n",
        "\n",
        "\n",
        "x = transform_image(image)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "synset is used to transform the label from number of ImageNet class to\n",
        "the word human can understand.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "synset_url = \"\".join(\n",
        "    [\n",
        "        \"https://gist.githubusercontent.com/zhreshold/\",\n",
        "        \"4d0b62f3d01426887599d4f7ede23ee5/raw/\",\n",
        "        \"596b27d23537e5a1b5751d2b0481ef172f58b539/\",\n",
        "        \"imagenet1000_clsid_to_human.txt\",\n",
        "    ]\n",
        ")\n",
        "synset_name = \"imagenet1000_clsid_to_human.txt\"\n",
        "synset_path = download_testdata(synset_url, synset_name, module=\"data\")\n",
        "with open(synset_path) as f:\n",
        "    synset = eval(f.read())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now we would like to port the Gluon model to a portable computational graph.\n",
        "It's as easy as several lines.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "# We support MXNet static graph(symbol) and HybridBlock in mxnet.gluon\n",
        "shape_dict = {\"data\": x.shape}\n",
        "mod, params = relay.frontend.from_mxnet(block, shape_dict)\n",
        "# we want a probability so add a softmax operator\n",
        "func = mod[\"main\"]\n",
        "func = relay.Function(func.params, relay.nn.softmax(func.body), None, func.type_params, func.attrs)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Here are some basic data workload configurations.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "batch_size = 1\n",
        "num_classes = 1000\n",
        "image_shape = (3, 224, 224)\n",
        "data_shape = (batch_size,) + image_shape"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Compile The Graph\n",
        "To compile the graph, we call the :py:func:`relay.build` function\n",
        "with the graph configuration and parameters. However, You cannot to\n",
        "deploy a x86 program on a device with ARM instruction set. It means\n",
        "Relay also needs to know the compilation option of target device,\n",
        "apart from arguments :code:`net` and :code:`params` to specify the\n",
        "deep learning workload. Actually, the option matters, different option\n",
        "will lead to very different performance.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "If we run the example on our x86 server for demonstration, we can simply\n",
        "set it as :code:`llvm`. If running it on the Raspberry Pi, we need to\n",
        "specify its instruction set. Set :code:`local_demo` to False if you want\n",
        "to run this tutorial with a real device.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "local_demo = True\n",
        "\n",
        "if local_demo:\n",
        "    target = tvm.target.Target(\"llvm\")\n",
        "else:\n",
        "    target = tvm.target.arm_cpu(\"rasp3b\")\n",
        "    # The above line is a simple form of\n",
        "    # target = tvm.target.Target('llvm -device=arm_cpu -model=bcm2837 -mtriple=armv7l-linux-gnueabihf -mattr=+neon')\n",
        "\n",
        "with tvm.transform.PassContext(opt_level=3):\n",
        "    lib = relay.build(func, target, params=params)\n",
        "\n",
        "# After `relay.build`, you will get three return values: graph,\n",
        "# library and the new parameter, since we do some optimization that will\n",
        "# change the parameters but keep the result of model as the same.\n",
        "\n",
        "# Save the library at local temporary directory.\n",
        "tmp = utils.tempdir()\n",
        "lib_fname = tmp.relpath(\"net.tar\")\n",
        "lib.export_library(lib_fname)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Deploy the Model Remotely by RPC\n",
        "With RPC, you can deploy the model remotely from your host machine\n",
        "to the remote device.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "# obtain an RPC session from remote device.\n",
        "if local_demo:\n",
        "    remote = rpc.LocalSession()\n",
        "else:\n",
        "    # The following is my environment, change this to the IP address of your target device\n",
        "    host = \"10.77.1.162\"\n",
        "    port = 9090\n",
        "    remote = rpc.connect(host, port)\n",
        "\n",
        "# upload the library to remote device and load it\n",
        "remote.upload(lib_fname)\n",
        "rlib = remote.load_module(\"net.tar\")\n",
        "\n",
        "# create the remote runtime module\n",
        "dev = remote.cpu(0)\n",
        "module = runtime.GraphModule(rlib[\"default\"](dev))\n",
        "# set input data\n",
        "module.set_input(\"data\", tvm.nd.array(x.astype(\"float32\")))\n",
        "# run\n",
        "module.run()\n",
        "# get output\n",
        "out = module.get_output(0)\n",
        "# get top1 result\n",
        "top1 = np.argmax(out.numpy())\n",
        "print(\"TVM prediction top-1: {}\".format(synset[top1]))"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.16"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
